Image processing apparatus, image processing method, image processing program, and recording medium

ABSTRACT

Activities are normalized corresponding to a characteristic of a pattern of a frame. The picture quality is optimized by adaptive quantization. A normalized activity norm_act [m] of each macro block is obtained with an average activity avg_act of the frame and the activity act [m] of each macro block according to the following formulas (1) and (2). In the following formulas, att is a parameter, for example att=0.125.
 
norm_gain=att×avg_act +1   (1)
 
norm_act[ m ]={(norm_gain×act[ m ]×act[ m] )+(avg_act×avg_act)}÷{(act[ m ]×act[ m ])+(norm_gain×avg_act×avg_act)}  (2)
 
     Thus, the normalized activities norm_act [m] are normalized in the range from 1/norm_gain to norm_gain. Since norm_gain is proportional to the average activity avg_act, when a picture has a flat pattern whose average activity avg_act is small, the normalizing range becomes small. Thus, the quantized values of macro blocks are not largely different. As a result, the picture is equally quantized. In contrast, when a picture has a complex pattern whose average activity avg_act is large, the normalizing range becomes large. Thus, the quantized values of macro block are largely different. As a result, a macro block having a flat pattern is finely quantized, whereas a macro block having a complex pattern is coarsely quantized.

TECHNICAL FIELD

The present invention relates to a picture processing apparatus, apicture processing method, a picture processing program, and a recordingmedium for controlling a generated code amount in compression encodingfor a picture signal using quantization of each block of the picturesignal so that the code amount of each frame does not exceed apredetermined amount.

BACKGROUND ART

In a conventional compression-encoding system that compresses andencodes picture data, picture data is quantized in the unit of one blockcomposed of a predetermined number of pixels. For example, in the MPEG2(Moving Pictures Experts Group 2), such a compression-encoding system isused. In the MPEG2, picture data is transformed in the DCT (DiscreteCosine Transform) method in the unit of one block composed of apredetermined number of pixels. The obtained DCT coefficients arequantized. As a result, the picture data is compression-encoded. In theMPEG2, with a quantizer step designated with a quantizer scale, DCTcoefficients are compressed and encoded.

Conventionally, in the MPEG2 compression-encoding, a method forcalculating an index referred to as activity that represents complexityor smoothness of a picture to be compressed and optimizing the picturequality using adaptive quantization corresponding to the activity isknown.

This method is performed as follows. When the compressing process isperformed for a simple and smooth picture region, the deterioration ofthe picture quality is obstructive. In this region (referred to as flatregion), the picture data is finely quantized with a quantizer scalewhose quantizer step is small. In contrast, when the compressing processis performed for a complex picture region, the deterioration of thepicture quality is not obstructive. In this region, the picture data iscoarsely quantized with a quantizer scale whose quantizer step is large.Thus, with a limited code amount, the picture quality can be effectivelyoptimized.

When picture data is compressed, as was described above, each pictureregion is divided into pixel blocks each having a predetermined size.The picture data is quantized and transformed using the DCT method inthe unit of one pixel block. The MPEG2 standard prescribes a block ofeight pixels×eight lines as the minimum process unit. In addition, theMPEG2 standard prescribes that each block of eight pixels×eight linesshould be transformed using the DCT method and that the resultant DCTcoefficients should be quantized in the unit of one macro block of 16pixels×16 lines.

On the other hand, although the MPEG2 standard does not clearlyprescribe a unit for calculating the forgoing activity, however, theMPEG2 TM5 (Test Model 5) has proposed that the activity should beprocessed in the unit of one sub block of eight pixels×eight lines,which is the same as one DCT block.

Next, the activity calculating method in “adaptive quantizationconsidering visual characteristic” that has been used in the MPEG2 TM5will be described.

The adaptive quantization is to vary a quantizer scale Qj correspondingto the state of a picture with an activity of each macro block so as tocontrol a generated code amount of for example one frame and improve thepicture quality. The quantizer scale Qj is varied with an activity sothat in a flat region, where the deterioration of the picture quality isobstructive, the picture data is quantized with a quantizer scale Qjwhose quantizer step is small and in a complex picture region, where thedeterioration of the picture quality is not obstructive, the picturedata is quantized with a quantizer scale Qj whose quantizer step islarge.

An activity is calculated with pixel values of a luminance signal of anoriginal picture rather than a predictive error. In other words, anactivity is calculated with pixel values of a total of eight blocks thatare four blocks in the frame DCT encoding mode and our blocks in thefield DCT encoding mode. For example, an activity act_(j) of a j-thmacro block is obtained by calculating the following Formulas (1) to (3)in the reverse order (namely, in the order of Formula (3), Formula (2),and Formula (1)).act_(j)=1+min[sblk=1, 8](var_sblk)  (1)var_sblk=1/64 Σ[k=1, 64](Pk−Pavg)²  (2)Pavg=1/64 Σ[k=1, 64]P _(k)  (3)

where P_(k) is a pixel value of a block of a luminance signal of theoriginal picture. In Formula (3), 64 pixel values of a block of 8×8 aresummed and the result is divided by 64. As a result, an average valuePavg of pixel values P_(k) of the block is obtained. Next, in Formula(2), the difference between the average value Pavg and each pixel valueP_(k) is obtained. As a result, average difference values var_sblk ofthe block of 8×8 are calculated. In Formula (1), with the minimum valueof the average difference values var_sblk, an activity act_(j) of thej-th macro block is obtained. The minimum value is used because even ifa part of the macro block contains a flat portion, it is necessary tofinely quantize the macro block.

In the MPEG2 TM5, a normalized activity Nact_(j) that has values in therange from “2.0” to “0.5” is obtained from activities act_(j) of macroblocks corresponding to the following Formula (4).Nact_(j)=(2×act_(j)+avg_act)/(act_(j)+2×avg_act)  (4)

where “avg_act” represents an average value (average activity) ofactivities act_(j) of an encoded frame immediately preceded by a frame(picture) that is currently being processed.

A quantizer scale mquant_(j) that considers a visual characteristic isgiven by the following Formula (5) corresponding to a quantizer scaleQ_(j) that is obtained for controlling a generated code amount of oneframe.mquant_(j) =Q _(j)×Nact_(j)  (5)

When each macro block is quantized with such a quantizer scalemquant_(j), while the code amount of one whole frame is kept in apredetermined range, each macro block is optimally quantizedcorresponding to flatness and complexity of a picture of the frame. As aresult, while a limited code amount is effectively used, the picture iseffectively compressed with much suppression of the picture quality.

When normalization is performed according to the MPEG2 TM5, a normalizedactivity Nact_(j) is obtained by the forgoing Formula (4). In otherwords, whatever pattern a frame has, the normalization ranges from “0.5”to “2.0”. Thus, in a frame having for example a flat pattern, quantizedvalues of individual macro blocks largely vary.

On the other hand, in a frame having a complex pattern, a macro block ofa complex portion uses a lat of a code amount. Thus, a macro block of arelatively flat pattern portion uses a limited code amount. As a result,the picture quality of a macro block having a flat pattern whosedeterioration tends to be obstructive deteriorates.

The deterioration of the picture quality becomes remarkable especiallywhen the compression rate is low and the code amount is relativelyaffordable.

Therefore, an object of the present invention is to provide a pictureprocessing apparatus, a picture processing method, a picture processingprogram, and a recording medium for normalizing an activitycorresponding to a characteristic of a pattern of a frame and optimizingthe picture quality using adaptive quantization.

DISCLOSURE OF THE INVENTION

To solve the forgoing problem, the present invention is a pictureprocessing apparatus, comprising average activity calculating means forcalculating an average activity with picture data; normalized activitycalculating means for dynamically designating a normalizing rangecorresponding to the average activity calculated by the average activitycalculating means and calculating a normalized activity with the averageactivity in the normalizing range; and quantizing means for quantizingthe picture data with the normalized activity calculated by thenormalized activity calculating means.

In addition, the present invention is a picture processing method,comprising the steps of calculating an average activity with picturedata; dynamically designating a normalizing range corresponding to theaverage activity calculated at the average activity calculating step andcalculating a normalized activity with the average activity in thenormalizing range; and quantizing the picture data with the normalizedactivity calculated at the normalized activity calculating step.

In addition, the present invention is a picture processing program forcausing a computer apparatus to execute a picture processing method forquantizing picture data, the picture processing method comprising thesteps of calculating an average activity with picture data; dynamicallydesignating a normalizing range corresponding to the average activitycalculated at the average activity calculating step and calculating anormalized activity with the average activity in the normalizing range;and quantizing the picture data with the normalized activity calculatedat the normalized activity calculating step.

In addition, the present invention is a recording medium on whichpicture processing program for causing a computer apparatus to execute apicture processing method for quantizing picture data has been recorded,the picture processing method comprising the steps of calculating anaverage activity with picture data; dynamically designating anormalizing range corresponding to the average activity calculated atthe average activity calculating step and calculating a normalizedactivity with the average activity in the normalizing range; andquantizing the picture data with the normalized activity calculated atthe normalized activity calculating step.

As was described above, according to the present invention, anormalizing range is dynamically designated corresponding to an averageactivity calculated with input picture data. Corresponding to thenormalizing range, with the average activity, a normalized activity iscalculated. With the calculated normalized activity, picture data isquantized. Thus, picture data can be quantized corresponding to acharacteristic thereof.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a graph showing an example of a characteristic of a formulathat expresses normalization of a second example;

FIG. 2 is a graph showing an example of a characteristic ofnormalization according to the MPEG TM5;

FIG. 3 is a schematic diagram showing an example of a flat pattern;

FIG. 4 is a schematic diagram showing an example that contains both acomplex pattern and a flat pattern;

FIG. 5 is a schematic diagram showing an example of a normalizedactivity of which a picture of a flat pattern is normalized according tothe MPEG TM5;

FIG. 6 is a schematic diagram showing an example of a normalizedactivity of which a picture wholly composed of a flat pattern isnormalized according to the present invention;

FIG. 7 is a schematic diagram showing an example of a normalizedactivity of which a picture having both a complex pattern and a flatpattern is normalized according to the MPEG TM5;

FIG. 8 is a schematic diagram showing an example of a normalizedactivity of which a picture having both a complex pattern and a flatpattern is normalized according to the present invention;

FIG. 9A and FIG. 9B are a block diagram showing an example of astructure of a digital VTR according to an embodiment of the presentinvention;

FIG. 10A, FIG. 10B, and FIG. 10C are a block diagram more practicallyshowing an example of a structure of an MPEG encoder;

FIG. 11A, FIG. 11B, and FIG. 11C are schematic diagrams showing examplesof structures of streams transferred by individual portions of the MPEGencoder;

FIG. 12A, FIG. 12B, and FIG. 12C are schematic diagrams showing examplesof structures of streams transferred by the individual portions of theMPEG encoder;

FIG. 13A and FIG. 13B are schematic diagrams showing examples ofstructures of streams transferred by the individual portions of the MPEGencoder;

FIG. 14A and FIG. 14B are schematic diagrams showing examples ofstructures of streams transferred by the individual portions of the MPEGencoder;

FIG. 15A, FIG. 15B, and FIG. 15C are schematic diagrams showing examplesof structures of streams transferred by the individual portions of theMPEG encoder;

FIG. 16A, FIG. 16B, and FIG. 16C are schematic diagrams showing examplesof structures of streams transferred by the individual portions of theMPEG encoder;

FIG. 17 is a schematic diagram showing an example of structure of astream transferred by the individual portions of the MPEG encoder;

FIG. 18 is a block diagram showing an example of a structure forcalculating a normalized activity according to the present invention;and

FIG. 19 is a flow chart showing an example of which a process of theMPEG encoder is implemented by software.

BEST MODES FOR CARRYING OUT THE INVENTION

Next, an embodiment of the present invention will be described.According to the present invention, a normalized activity in a targetrange is obtained by dynamically varying a normalizing rangecorresponding to a characteristic of a picture in the target range, forexample, flatness and complexity in the target range, in other words, anaverage activity. Corresponding to the normalized activity, the picturein the target range is quantized.

When a frame is a target range, if the whole frame is a flat pattern,the normalizing range of activities is narrowed. Thus, with activitiesthat are equal on the whole picture, it can be quantized. As a result,with equal flatness on the whole picture, a high picture quality can beobtained. In contrast, when one frame is a picture that has both acomplex region and a flat region, the normalizing range of activities iswidened. As a result, a large code amount is assigned to a flat regionin which noise is obstructive. In contrast, a complex region in whichnoise is not obstructive is coarsely quantized. As a result, a picturethat has visibly high quality can be obtained.

More practically, as a first example, when an average activity of oneframe is denoted by avg_act and an activity of a macro block having amacro block address m is denoted by act [m], a normalized activitynorm_act [m] is calculated corresponding to the following Formula (6)and Formula (7).norm_gain=att×avg_act+1  (6)norm_act[m]={norm_gain×act[m]+avg_act}÷{act [m]+norm_gain×avg_act}  (7)

In Formula (6) and Formula (7), when both act [m] and avg_act become 0,since the denominator becomes 0, it is treated as norm_act [m]=1. Inaddition, att is a parameter. For example, att=0.125 is assigned.

In Formula (7), a value norm_gain is substituted for coefficient “2” ofFormula (4) in the normalized activity calculating method according tothe MPEG TM5. In Formula (7), with activity act [m]=0, the relation ofnormalized activity norm_act [m]=1/norm_gain is obtained. On the otherhand, when an activity act [m] is much larger than the average activityavg_act, a normalized activity norm_act [m] becomes close to the valuenorm_gain. In other words, a normalized activity norm_act [m] of eachmacro block is normalized in the range from 1/norm_gain to norm_gain.

As expressed in Formula (6), the value norm_gain is proportional to theaverage activity avg_act. Thus, the normalizing range of a frame havinga flat pattern, whose average activity avg_act is small, is narrow. As aresult, without a large difference of quantized values of macro blocks,the frame is equally quantized. In contrast, the normalizing range of aframe that has a complex pattern, whose average activity is large,becomes large. Thus, the difference of quantized values of macro blocksbecomes large. Thus, a macro block having a flat pattern is finelyquantized, whereas a macro block having a complex pattern is coarselyquantized.

As a second example, an expression of which the activity act [m] on thedividend side of Formula (7) of the first example is squared and theaverage activity avg_act on the divisor side thereof is squared may beused. The calculating method for a normalized activity norm_act [m] of amacro block is expressed by Formula (8) and Formula (9). The meanings ofterms of Formula (8) and Formula (9) are the same as those of Formula(6) and (7).norm_gain=att×avg_act+1  (8)norm_act[m]={(norm_gain×act[m]×act[m])+(avg_act×avg_act)}÷{(act[m]×act[m])+(norm_gain×avg_act×avg_act)}  (9)

In Formula (8) and Formula (9), when both act [m] and avg_act become 0,since the denominators thereof become 0, it is assumed that norm_act[m]=1. In addition, att is a parameter. In this example, att=0.125 isassigned.

Like in Formula (6) and Formula (7) of the first example, in Formula (8)and Formula (9) the second example, a normalized activity norm_act [m]of a macro block is in the range from 1/norm_gain to norm_gain. Thus,like in the first example, when a picture has a flat pattern whoseaverage activity avg_act is small, the picture is equally quantized.When a picture has a complex pattern whose average activity avg_act islarge, quantized values of macro blocks become large.

In the second example, each of the activity act [m] on the dividend sideand the average activity avg_act [m] on the divisor side is squaredunlike with the first example. Thus, as the average activity avg_act andan activity act [m] of a macro block vary, the difference of thenormalized activities norm_act [m] becomes larger than those of thefirst example. Consequently, a more preferred result can be obtainedfrom the second example than the first example. Next, focused on thesecond example, the normalization will be described.

FIG. 1 is a graph showing an example of a characteristic of Formula (8)and Formula (9), which express the normalization of the second example.FIG. 2 is a graph showing an example of a characteristic of Formula (4),which expresses the normalization according to the MPEG TM5 described inthe section of the Related Art. In FIG. 1 and FIG. 2, X axis, Y axis,and Z axis represent average activity avg_act, activity act [m], andnormalized activity norm_act [m], respectively. Ranges “woman”, “noise”,“sheep”, and “lawn” shown in FIG. 1 and FIG. 2 represent ranges ofnormalized activities norm_act [m] of corresponding portions of samplepictures shown in FIG. 3 and FIG. 4.

FIG. 3 shows an example of a picture wholly composed of a flat pattern.In particular, a pattern of the background of the picture is composed ofa figure painted with one or several colors. It can be said that thepicture has a simple structure. The “lawn” and “sheep” portions areparts of the background. FIG. 4 shows an example of a picture that hasboth a complex pattern and a flat pattern. In FIG. 4, the “woman”portion is hemmed at a center portion. The “noise” portion surrounds thehemmed “woman” portion. As in the picture shown in FIG. 3, the “woman”portion has a relatively flat pattern. However, the “noise” portionaround the “woman” portion has a low correlation with an adjacentportion in the level of pixels. Thus, the “noise” portion has a complexpattern.

In FIG. 1, when the average activity avg_act is “0”, the normalizedactivity norm_act is 1. When the average activity avg_act is 16,norm_gain becomes 3. Thus, it is clear that the normalized activitynorm_act [m] distributes in the range from ⅓<norm_act [m]<3. In FIG. 1,norm_gain is denoted by rate.

Since FIG. 2 shows only 64 activities act [m], the range of the averageactivity is not clear. However, whatever the average activity avg_acthas, it is clear that the normalized activity norm_act [m] distributesin the normalized range of ½<norm_act [m]<2.

When the pictures shown in FIG. 1 and FIG. 2 are compared, it is clearthat a wide range is assigned to the normalized activity norm_act [m]for “noise”, which is a complex pattern, whereas a narrow range isassigned to the normalized activity norm_act [m] for “woman”, which is arelatively flat pattern. In contrast, in FIG. 2 according to the MPEGTM5, a similar range is assigned to the normalized activity norm_act [m]for both “noise” and “woman”. On the other hand, the range of thenormalized activity norm_act [m] assigned for the flat patterns “sheep”and “lawn” shown in FIG. 1 according to the present invention is smallerthan that shown in FIG. 2 according to the MPEG TM5.

FIG. 5 to FIG. 8 show distributions of the normalized activitiesnorm_act [m] according to the MPEG TM5 and of the normalized activitiesnorm_act [m] according to the present invention for the sample picturesshown in FIG. 3 and FIG. 4. In FIG. 5 to FIG. 8, diameters of “● (blackcircles)” relatively represent values of the normalized activitiesnorm_act [m] at corresponding positions (macro blocks) of the pictures.FIG. 5 and FIG. 6 correspond to FIG. 3, whereas FIG. 7 and FIG. 8correspond to FIG. 4.

The average activity avg_act of the picture shown in FIG. 3 is around 3.On the other hand, the average activity avg_act of the picture shown inFIG. 4 is around 16.

FIG. 5 shows an example of normalized activities norm_act [m] for thepicture shown in FIG. 3 that is normalized according to the conventionalMPEG TM5. In FIG. 5, a normalized activity norm_act [m] of a macro blockhaving a flat picture (a non-character portion of a calendar and aforeside portion of a train) is small. on the other hand, a normalizedactivity norm_act [m] of a macro block having a complex picture (forexample, a picture portion of a calendar and a tree portion at an upperleft portion) is large. As the whole picture, it is clear that thenormalized activities norm_act [m] largely vary with individual macroblocks.

Especially when the compression rate is low and the code amount isrelatively affordable, the quantized values of macro blocks largelyvary. Thus, as the picture quality of a whole picture becomes high, thedifference of picture quality of macro blocks can be clearly detected.

FIG. 6 shows an example of normalized activities norm_act [m] for thepicture shown in FIG. 3 that is normalized according to the presentinvention. It is clear that the deviation of sizes of “(black circles)”is small and that the values of the normalized activities norm_act [m]are relatively equal in the whole picture in comparison with the pictureshown in FIG. 5. In other words, it is clear that the value of anormalized activity norm_act [m] of a macro block having a flat pictureis not largely different from that of a macro block having a complexpicture in comparison with the example of the picture shown in FIG. 5.In the adaptive quantization, since macro blocks are quantized withquantized values proportional to values of the normalized activitiesnorm_act [m]. Thus, the whole picture is relatively equally quantized.

Next, an example of which the average activity avg_act is large as shownin FIG. 4 will be described. FIG. 7 shows an example of normalizedactivities norm_act [m] for the picture shown in FIG. 4 normalizedaccording to the MPEG TM5 as the related art. In FIG. 7, the rates ofdiameters of “● (black circles)” that represent the values of normalizedactivities norm_act [m] are different from those shown in FIG. 5.

As in FIG. 5, in FIG. 7, the value of a normalized activity norm_act [m]of a macro block having a flat picture (for example, a “woman” portion)is small, whereas the value of a normalized activity norm_act [m] of amacro block having a complex picture (a “noise” portion) is large. Thus,it is clear that the values of the normalized activities norm_act [m]vary with macro blocks.

However, in the example shown in FIG. 7, a macro block that has a verycomplex picture such as the “noise” portion uses a lot of a code amount.As a result, the code amount of a macro block that has a flat picture isrestricted. Thus, the picture quality of a macro block that has a flatpicture whose visual deterioration tends to be obstructive deteriorates.

FIG. 8 shows an example of normalized activities norm_act [m] for thepicture shown in FIG. 4 normalized according to the present invention.According to the present invention, the normalizing range of normalizedactivities norm_act [m] is dynamically varied corresponding to theaverage activity avg_act of the picture. Thus, when the value of theaverage activity avg_act is large because the picture is complex (in theexample shown in FIG. 4, act_avg≈16), the range of the normalizedactivities norm_act [m] is for example ⅓<norm_act [m]<3, which is largerthan that for a picture whose average activity avg_act is small.

Thus, the value of a normalized activity norm_act [m] of a macro blockthat has a flat picture is small. In contrast, the value of a normalizedactivity norm_act [m] of a macro block having a very complex picturesuch as a noise portion is large. In the example shown in FIG. 8, it isclear that the value of a normalized activity norm_act [m] of a portionhaving a flat picture is largely different from that of a portion havinga very complex picture.

Thus, according to the present invention, the adaptive quantization isvaried corresponding to a characteristic such as a pattern of a picture.As a result, an optimum picture quality corresponding to acharacteristic of a picture can be obtained.

Next, an example of a real structure according to an embodiment of thepresent invention will be described. FIG. 9A and FIG. 9B show an exampleof a structure of a digital VTR according to an embodiment of thepresent invention. The digital VTR can directly record a digital videosignal that has been compressed and encoded corresponding to the MPEGsystem onto a recording medium.

First of all, the structure and the processing operation of a recordingsystem of the digital VTR will be described. Signals that are input fromthe outside to the recording system are two types of serial digitalinterface signals that are an SDI (Serial Data Interface) signal and anSDTI (Serial Data Transport Interface) signal, an analog interfacesignal, and an external reference signal REF that is a control signal.

The SDI is an interface prescribed by the SMPTE so as to transmit a(4:2:2) component video signal, a digital audio signal, and additionaldata. The SDTI is an interface through which an MPEG elementary stream(referred to as MPEG ES) that is a stream of which a digital videosignal has been compression-encoded according to the MPEG system. The ESis 4:2:2 components. As described above, the ES is a stream of all Ipictures having the relation of 1 GOP=1 picture. In the SDTI-CP (ContentPackage) format, the MPEG ES is separated into access units and packedto a packet in the unit of one frame. The SDTI-CP uses a sufficienttransmission band (clock rate: 27 MHz or 36 MHz or stream bit rate: 270Mbps or 360 Mbps). In one frame period, the ES can be transmitted as aburst.

The SDI signal transmitted through the SDI is input to an SDI inputportion 101. An analog input signal as an analog video signal is inputto an analog input portion 120. The analog input portion 120 convertsthe analog input signal into a digital signal, maps the digital signalto for example the aforementioned SDI format, and outputs the resultantSDI signal. The SDI signal, where the analog input signal has beenconverted and mapped to the SDI format, is supplied to the SDI inputportion 101.

The SDI input portion 101 converts the supplied SDI signal as a serialsignal into a parallel signal. In addition, the SDI input portion 101extracts an input synchronous signal as an input phase reference fromthe SDI signal and outputs the extracted input synchronous signal to atiming generator TG 102.

In addition, the SDI input portion 101 separates a video signal and anaudio signal from the converted parallel signal. The separated videoinput signal and audio signal are output to an MPEG encoder 103 and adelay circuit 104, respectively.

The timing generator TG 102 extracts a reference synchronous signal froman input external reference signal REF. In synchronization with adesignated one of the reference synchronous signal or the inputsynchronous signal, which has been supplied from the SDI input portion101, the timing generator TG 102 generates a timing signal necessary forthe digital VTR and supplies it as timing pulses to each block.

The MPEG encoder 103 converts the input video signal into coefficientdata according to the DCT method, quantizes the coefficient data, andencodes the quantized data with a variable length code.Variable-length-coded (VLC) data, which has been output from the MPEGencoder 103, is an elementary stream (ES) according to the MPEG2. Theelementary stream, which has been output from the MPEG encoder 103, issupplied to one of input terminals of a recording side multi-formatconverter (hereinafter referred to as recording side MFC).

The delay circuit 104 functions as a delay line that delays the inputaudio signal as non-compressed signal corresponding to the delay of thevideo signal of the MPEG encoder 103. The audio signal, which has beendelayed by the delay circuit 104, is output to an ECC encoder 107. Thisis because the digital VTR according to the embodiment of the presentinvention treats the audio signal as a non-compressed signal.

The SDTI signal supplied from the outside through the SDTI is input toan SDTI input portion 105. The SDTI input portion 105 detects thesynchronization of the SDTI signal. The SDTI signal is temporarilybuffered and the elementary stream is extracted therefrom. The extractedelementary stream is supplied to the other input terminal of therecording side MFC 106. The synchronous signal, which has beensynchronously detected, is supplied to the timing generator TG 102 (notshown).

The SDTI input portion 105 extracts a digital audio signal from theinput SDTI signal. The extracted digital audio signal is supplied to theECC encoder 107.

In the digital VTR according to the present invention, an MPEG ES can bedirectly input independent from the base band video signal, which hasbeen input from the SDI input portion 101.

The recording side MFC 106 has a stream converter and a selector. Therecording side MFC 106 selects one of the MPEG ES supplied from the SDIinput portion 101 and the MPEG ES supplied from the SDTI input portion105, and collects DCT coefficients of DCT blocks of one macro block sothat frequency components are rearranged in the order of frequencycomponents. The resultant stream, of which the coefficients of the MPEGES have been rearranged, is referred to as converted elementary stream.Since the MPEG ES is rearranged, when a search reproduction isperformed, as many DC coefficients and low order AC coefficients can becollected as possible, which contributes the improvement of the qualityof the search picture. The converted elementary stream is supplied tothe ECC encoder 107.

A main memory (not shown) having a large storage capacity is connectedto the ECC encoder 107. The ECC encoder 107 has a packing and shufflingportion, an audio outer code encoder, a video outer code encoder, aninner code encoder, an audio shuffling portion, a video shufflingportion, and so forth. In addition, the ECC encoder 107 has an syncblock ID adding circuit and a synchronous signal adding circuit.According to the first embodiment of the present invention, as an errorcorrection code for the video signal and audio signal, a product code isused. In the product code, a data symbol is dually encoded in such amanner that the two-dimensional array of the video signal or audiosignal is encoded in the vertical direction with an outer code and thatthe two-dimensional array is encoded in the horizontal direction with aninner code. As the outer code and inner code, the Reed-Solomon code canbe used.

The converted elementary stream, which has been output from therecording side MFC 106, is supplied to the ECC encoder 107. In addition,the audio signals, which are output from the SDTI input portion 105 andthe delay circuit 104, are supplied to the ECC encoder 107. The ECCencoder 107 shuffles the converted elementary stream and audio signals,encodes them with an error correction code, adds IDs and a synchronoussignal to sync blocks, and outputs the resultant signal as record data.

The record data, which has been output from the ECC encoder 107, isconverted into a record RF signal by the equalizer EQ 108 that has arecording amplifier. The record RF signal is supplied to a rotating dram109. The rotating dram 109 records the record signal on a magnetic tape110. The rotating dram 109 has a rotating head disposed in apredetermined manner. In reality, a plurality of magnetic heads whoseazimuths are different from each other and that form adjacent tracks aredisposed.

When necessary, the record data may be scrambled. When data is recorded,it may be digitally modulated. In addition, the partial response class 4and Viterbi code may be used. The equalizer EQ 108 has both a recordingside structure and a reproducing side structure.

Next, the structure and the processing operation of the reproducingsystem of the digital VTR will be described. In the reproduction mode, areproduction signal that is reproduced from the magnetic tape 110 by therotating dram 109 is supplied to the reproducing side structure of theequalizer EQ 108 that has a reproducing amplifier and so forth. Theequalizer EQ 108 equalizes the reproduction signal and trims the waveshape thereof. When necessary, the equalizer EQ 108 demodulates thedigital modulation and decodes the Viterbi code. An output of theequalizer EQ 108 is supplied to an ECC decoder 111.

The ECC decoder 111 performs the reverse process of the ECC encoder 107.The ECC decoder 111 has a main memory having a large storage capacity,an inner code decoder, an audio deshuffling portion, a video deshufflingportion, and an outer code decoder. In addition, the ECC decoder 111 hasa deshuffling and depacking portion and a data interpolating portion forvideo data. Likewise, the ECC decoder 111 has an audio AUX separatingportion and a data interpolating portion for audio data.

The ECC decoder 111 detects the synchronization of reproduction data. Inother words, the ECC decoder 111 detects a synchronous signal added atthe beginning of a sync block and extracts the sync block from thereproduction signal. The ECC decoder 111 corrects an error of each syncblock of the reproduction data with an inner code. Thereafter, the ECCdecoder 111 performs an ID interpolating process for each sync block.The ECC decoder 111 separates video data and audio data from thereproduction data, where IDs have been interpolated. The ECC decoder 111deshuffles the video data and audio data so that they are restored tothe original data. The ECC decoder 111 corrects an error of thedeshuffled data with an outer code.

When the ECC decoder 111 cannot correct an error of data, which exceedsits error correcting performance, the ECC decoder 111 sets an error flagto the data. For an error of video data, the ECC decoder 111 outputs asignal ERR that represents that data has an error.

The reproduction audio data, whose error has been corrected, is suppliedto an SDTI output portion 115. A delay circuit 114 delays thereproduction audio data for a predetermined amount and supplies thedelayed reproduction audio data to an SDI output portion 116. The delaycircuit 114 absorbs the delay of the video data processed in an MPEGdecoder 113 that will be described later.

On the other hand, the video data, whose error has been corrected, issupplied as converted reproduction elementary stream to a reproducingside MFC circuit 112. The aforementioned signal ERR is also supplied tothe reproducing side MFC circuit 112. The reproducing side MFC circuit112 performs the reverse process of the recording side MFC 106. Thereproducing side MFC circuit 112 has a stream converter. The streamconverter performs the reverse process of the recording side streamconverter. In other words, the stream converter rearranges DCTcoefficients arranged in the order of frequency components to thosearranged in the order of DCT blocks. As a result, the reproductionsignal is converted into an elementary stream according to the MPEG2. Atthat point, when the signal ERR is supplied from the ECC decoder 111 tothe reproducing side MFC circuit 112, it replaces the corresponding datawith a signal that perfectly complies with the MPEG2.

The MPEG ES, which has been output from the reproducing side MFC circuit112, is supplied to the MPEG decoder 113 and the SDTI output portion115. The MPEG decoder 113 decodes the supplied MPEG ES so as to restoreit to the original video signal, which has not been compressed. In otherwords, the MPEG decoder 113 performs an inversely quantizing process andan inverse DCT process for the supplied MPEG ES. The decoded videosignal is supplied to an SDI output portion 116.

As described above, the audio data, which had been separated from thevideo data by the ECC decoder 111, has been supplied to the SDI outputportion 116 through the delay circuit 114. The SDI output portion 116maps the supplied video data and audio data in the SDI format so as toconvert them into the SDI signal having the SDI data structure. The SDIsignal is output to the outside.

On the other hand, as described above, the audio data, which had beenseparated from the video data by the ECC decoder 111, has been suppliedto the SDTI output portion 115. The SDTI output portion 115 maps thevideo data and audio data as the supplied elementary stream in the SDTIformat so as to convert them into the SDTI signal having the SDTI datastructure. The SDTI signal is output to the outside.

A system controller 117 (abbreviated as sys-con 117 in FIG. 9A and FIG.9B) is composed of for example a micro computer. The system controller117 communicates with each block using a digital signal SY_IO so as tocontrol the entire operation of the digital VTR. A servo 118communicates with the system controller 117 using a signal SY_SV. Usingthe signal SV_IO, the servo 118 controls the traveling of the magnetictape 110 and drives and controls the rotating dram 109.

FIG. 10A, FIG. 10B, and FIG. 10C more practically show the structure ofthe forgoing example of the MPEG encoder. FIG. 11A, FIG. 11B, FIG. 11C,FIG. 12A, FIG. 12B, FIG. 12C, FIG. 13A, FIG. 13B, FIG. 14A, FIG. 14B,FIG. 15A, FIG. 15B, FIG. 15C, FIG. 16A, FIG. 16B, FIG. 16C, and FIG. 17show examples of the structures of streams transferred in the individualportions of FIG. 10A, FIG. 10B, and FIG. 10C.

The MPEG encoder 103 is composed of an input field activity averagingprocess portion 103A, a pre-encoding process portion 103B, and an encodeportion 103C. The input field activity averaging process portion 103Aobtains the average value of activities of the input video data andsupplies the obtained average value to the pre-encoding process portion103B. The pre-encoding process portion 103B estimates the generated codeamount of quantized input video data with the average value of theactivities. According to the estimated result, while controlling thecode amount, the encode portion 103C actually quantizes the input videodata, encodes the quantized video data with a variable length code, andoutputs the resultant data as an MPEG ES.

A timing generator TG 220 generates a timing signal necessary for theMPEG encoder 103 with a horizontal synchronous signal HD, a verticalsynchronous signal VD, and a field synchronous signal FLD that have beensupplied from for example the timing generator TG 102 shown in FIG. 9Aand FIG. 9B. A CPU I/F block 221 is an interface with the systemcontroller 117 shown in FIG. 9A and FIG. 9B. With a control signal anddata transferred through the CPU I/F block 221, the operation of theMPEG encoder 103 is controlled.

First of all, the process of the input field activity averaging processportion 103A will be described. Video data, which has been output fromthe SDI input portion 101 and input to the MPEG encoder 103, is suppliedto an input portion 201. The input field activity averaging processportion 103A converts the video data into data that can be stored in amain memory 203 and checks a parity for the video data. The video data,which has been output from the input portion 201, is supplied to aheader creating portion 202. Using a vertical blanking region or thelike, headers according to the MPEG, for example sequence_header,quantizer_matrix, and gop_header, are extracted. The extracted headersare stored in the main memory 203. These headers are designated mainlyby the CPU I/F block 221. In other than the vertical blanking region, inthe header creating portion 202, the video data supplied from the inputportion 201 is stored in the main memory 203.

The main memory 203 is a frame memory for a picture. The main memory 203rearranges video data and absorbs the system delay. The video data isrearranged by for example an address controller (not shown) thatcontrols read addresses of the main memory 203. In FIG. 8A, FIG. 8B, andFIG. 8C, 8 lines, 0.5 frame, and 1 frame in the block of the main memory203 represent delay values as read timings of the main memory 203. Theyare properly controlled corresponding to a command issued from thetiming generator TG 220.

A raster scan/block scan converting portion 204 extracts macro blocksaccording to the MPEG from each line of the video data, which has beenstored in the main memory 203, and supplies the extracted macro blocksto an activity portion 205 disposed downstream thereof. According to theembodiment, with only the first field, activities are calculated. Thus,the macro blocks that are output from the raster scan/block scanconverting portion 204 are composed of video data of the first field.

As shown in FIG. 11A, at the beginning of the stream that is output fromthe raster scan/block scan converting portion 204, address informationof the vertical and horizontal directions of the macro block are placed.The address information is followed by a blank area having apredetermined size, followed by picture data for one macro block.

The stream has a data length of 576 words each of which is composed offor example eight bits. The last 512 words (referred to as data portion)are assigned a picture data area for one macro block. The first 64 words(referred to as header portion) contain the aforementioned addressinformation of the macro block. The other portion is a blank area fordata and flags embedded by each portion disposed downstream of theraster scan/block scan converting portion 204.

A macro block according to the MPEG is a matrix of 16 pixels×16 lines.However, the input field activity averaging process portion 103Aperforms the process for obtaining activities with only the first field.Thus, when eight lines of the first field have been stored in the mainmemory 203, the process can be started. In reality, corresponding to acommand issued from the timing generator TG 220, the process is properlystarted.

The activity portion 205 calculates an activity of each macro block. Inthe input field activity averaging process portion 103A, an activity ofeach macro block of only a first field is calculated. The calculatedresult is output as a field activity signal field_act. The signalfield_act is supplied to the averaging portion 206. The signal field_actis cumulated for one field. As a result, an average value avg act isobtained. The average value avg_act is supplied to the activity portion209 of the pre-encoding process portion 103B. The activity portion 209performs a pre-encoding process with the average values av_act of thefirst and second fields.

Thus, after the average value avg_act of the activities of the firstfield has been obtained, a pre-encoding process can be performed withthe average value in consideration of adaptive quantization.

Next, the pre-encoding process portion 103B will be described. Theraster scan/block scan converting portion 207A performs basically thesame process as the forgoing raster scan/block scan converting portion204. However, since the raster scan/block scan converting portion 207Ais to perform a pre-encoding process for estimating a code amount, theraster scan/block scan converting portion 207A requires video data ofboth the first field and the second field. Thus, when eight lines of thesecond field have been stored in the main memory 203, the rasterscan/block scan converting portion 207A can form a macro block having asize of 16 pixels×16 lines, which is dealt with the MPEG. At that point,the raster scan/block scan converting portion 207A can start theprocess. In reality, the raster scan/block scan converting portion 207Aproperly starts the process corresponding to a command received from thetiming generator TG 220.

Video data that is output from the raster scan/block scan convertingportion 207A is supplied to the DCT mode portion 208. The DCT modeportion 208 decides to select a field DCT encoding mode or a frame DCTencoding mode to encode video data.

In this case, instead of actually encoding video data, the DCT modeportion 208 calculates the sum of absolute values of difference valuesof vertically adjacent pixels in both the DCT encoding mode and theframe DCT encoding mode, compares them, and selects the mode whosecalculated sum is smaller than the other. The selected result istemporarily placed as DCT mode type data that is a flag in the stream.The flag is transferred to each portion downstream of the DCT modeportion 208. As shown in FIG. 11B, the DCT mode type data dct_type isplaced on the rear end side of a blank area of the header portion.

The activity portion. 209 performs basically the same process as theforgoing activity portion 205. However, as was described above, theactivity portion 209 is to perform the pre-encoding process. Thus, theactivity portion 209 calculates an activity of each macro block usingdata of both the first field and the second field.

The activity portion 209 first obtains an activity act and places itafter a macro block address of the header portion as shown in FIG. 11C.Thereafter, with the activity act and the average value avg_act of thefield activity obtained from the forgoing averaging portion 206, theactivity portion 209 obtains a normalized activity according to thepresent invention corresponding to Formula (8) and Formula (9). Theactivity portion 209 has a structure for calculating a normalizedactivity norm_act according to the present invention. The structure willbe described later.

The obtained normalized activity norm_act is temporarily placed as aflag in the header portion of the stream as shown in FIG. 12A. The flagis transferred to the individual portions downstream of the averagingportion 206. The activity act is overwritten to the normalized activitydata norm_act in the stream.

An output of the activity portion 209 is supplied to a DCT portion 210A.The DCT portion 210A divides the supplied macro block into DCT blockseach of which is composed of eight pixels×eight lines, performs thetwo-dimensional DCT for each DCT block, and generates DCT coefficients.As shown in FIG. 12B, the DCT coefficients are placed in the dataportion of the stream and supplied to a quantizer table portion 211A.

The quantizer table portion 211A quantizes the DCT coefficients, whichhave been transformed by the DCT portion 210A, with a quantizer matrix(quantizer_matrix). As shown in FIG. 12C, the DCT coefficients, whichhave been quantized by the quantizer table portion 211A, are placed inthe data portion of the stream and then output. An output of thequantizer table portion 211A is supplied to a multi-staged quantizerportion composed of a plurality of Q_n (quantizer) portions 212, 212, .. . , VLC portions 213, 213, . . . , cumulating portions Σ 214, 214, . .. , and cumulating portions Σ 215, 215, . . . The DCT coefficientsquantized by the quantizer table portion 211A are quantized on multiplestages of the quantizer portions.

The Q_n portions 212, 212, . . . quantize the DCT coefficients withdifferent quantizer scales (quantizer_scale) Q. The values of thequantizer scales Q are prescribed in for example the MPEG2 standard. TheQ_n portions 212, 212, . . . are composed of for example 31 quantizersaccording to the standard. At that point, since n=31, the Q_n portions212, 212, . . . are a Q_(—)1 portion, a Q_(—)2 portion, . . . , and aQ_(—)31 portion. With the quantizer scales Qn, the Q_n portions 212quantize DCT coefficients with the quantizer scales Qn assigned theretoat a total of 31 steps. Hereinafter, the quantizer scale values of theQ_n portions 212, 212, . . . are denoted by the quantizer scales Qnvalues.

The Q_n portions 212, 212, . . . quantize DCT coefficients with theirquantizer scale Qn values. At that point, with the quantizer scalemqaunt, which is in consideration of the visual characteristic, andwhich has been obtained using mqaunt=Q_n×norm_act with the normalizedactivity data norm_act obtained by the activity portion 209, theadaptive quantization is performed.

The DCT coefficients adaptively quantized by the Q_n portions 212, 212,. . . with the quantizer scales Qn are placed in the data portion of thestream as shown in FIG. 13A and supplied to the VLC portions 213, 213, .. . The VLC portions 213, 213, . . . scan DCT coefficients for theindividual quantizer scales Qn according to for example the zigzagscanning method and encodes them with a variable length code withreference to a VLC table according to for example the two-dimensionalHuffman code.

The data, which has been encoded with the variable length code by theVLC portions 213, 213, . . . , is placed in the data portion of thestream as shown in FIG. 13B and then output. Outputs of the VLC portions213, 213, . . . are supplied to the corresponding cumulating portions Σ214, 214, . . . ,

The cumulating portions Σ 214, 214, . . . cumulate the generated codeamounts for each macro block. As described above, when 31 types ofquantizing devices are used, 31 types of generated code amounts areobtained for each macro block. As shown in FIG. 14A, the generated codeamounts cumulated by the cumulating portions Σ 214, 214, . . . areplaced in the header portion of the stream. In other words, thegenerated code amounts quantized by the Q_(—)1 portion 212 to Q_nportion 212 for each macro block are placed in the header portion of thestream. The data portion of the stream is deleted. The stream of eachmacro block is supplied to the main memory 203.

The generated code amounts for each macro block, which have been outputfrom the cumulating portions Σ 214, 214, . . . , are supplied to therespective cumulating portions Σ 215, 215, . . . The cumulating portionsΣ 215, 215, . . . select generated code amounts for each macro blockquantized with quantizer_scale (=mquant), in which the forgoing visualcharacteristic has been considered, from those obtained by thecumulating portions Σ 214, 214, . . . and cumulate them for one frame.

The generated code amounts (frame data rates) cumulated by thecumulating portions Σ 215, 215, . . . for the quantizer scales Qn aresupplied as an n-word stream as shown in FIG. 14B to a rate controllingportion 217. When 31 types of quantizing devices are used as theforgoing example, corresponding 31 types of generated code amounts foreach frame are obtained.

Next, a method for obtaining a generated code amount will be describedmore practically. For example, “generated code amount of Q_(—)4 portion212” can be obtained as follows.

For example, in the case

norm_act [1]=1.3

norm_act [2]=1.5

norm_act [3]=0.8

norm_act [4]=1.0

. . . ,mqaunt [1]=4×1.3=5.2

The generated code amount of the Q_(—)5 portion 212 is obtained from theheader portion of FIG. 14A.mqaunt [2]=4×1.5=6.0

The generated code amount of the Q_(—)6 portion 212 is obtained from theheader portion of FIG. 14A.mqaunt [3]=4×0.8=3.2

The generated code amount of the Q_(—)3 portion 212 is obtained from theheader portion of FIG. 14A.mqaunt [4]=4×1.0=4.0

The generated code amount of the Q_(—)4 portion 212 is obtained from theheader portion of FIG. 14A.

. . .

They are cumulated for one frame. They are performed for each of theQ_(—)1 portion 212 to the Q_n portion 212. As a result, the generatedcode amount for one frame is obtained.

Next, the encoding process portion 103C will be described. The encodingprocess portion 103C performs the final encoding process. As describedabove, the pre-encoding process portion 103B estimates the generatedcode amounts for one frame in various quantizing operations. Theencoding process portion 103C encodes data corresponding to thegenerated code amount, which has been estimated for one frame so thatthe generated code amount does not exceed the pre-designated targetgenerated code amount and outputs an MPEG ES.

The data used in the encoding process portion 103C has been stored inthe main memory 203. However, as described above, when the generatedcode amounts for one frame have been estimated in various quantizingoperations by the pre-encoding process portion 103B, the encodingprocess portion 103C can start the process. As described above, theprocess of each portion of the encoding process portion 103C can beproperly started corresponding to a command issued from the timinggenerator TG 220.

Video data that has been read from the main memory 203 is supplied to araster scan/block scan converting portion 207B. The raster scan/blockscan converting portion 207B performs a process similar to the processof the raster scan/block scan converting portion 207A and extracts amacro block of 16 pixels×16 lines from the video data. As shown in FIG.15A, the extracted macro block is placed in a data portion correspondingto the header portion shown in FIG. 14A and supplied to a DCT modeportion 216.

Like the DCT mode portion 208, the DCT mode portion 216 decides to usethe field DCT encoding mode or the frame DCT encoding mode to encodedata. At that point, the DCT mode portion 208 has decided the DCT modeportion 208 and temporarily placed the result as DCT type data dct_typin the stream (see FIG. 15A). The DCT mode portion 216 detects the DCTtype data dct_typ from the stream and switches to the field encodingmode or the frame encoding mode corresponding to the detected DCT typedata dct_typ. An output of the DCT mode portion 216 is shown in FIG.15B.

A macro block that has been output from the DCT mode portion 216 issupplied to a DCT portion 210B. Like the DCT portion 210A, the DCTportion 210B two-dimensionally transforms the macro block into DCTcoefficients in the unit of one DCT block of eight pixels×eight lines.As shown in FIG. 15C, the DCT coefficients, into which the macro blockhas been two-dimensionally transformed corresponding to thetwo-dimensional DCT method, are placed in the data portion of the streamand then output from the DCT portion 210B.

A quantizer table portion 211B can be structured in the same manner asthe forgoing quantizer table portion 211A. The quantizer table portion211B quantizes the DCT coefficients transformed by the DCT portion 210Bwith a quantizer matrix. As shown in FIG. 16A, the DCT coefficientsquantized by the quantizer table portion 211B are placed in the dataportion of the stream and supplied to a rate controlling portion 217.

The rate controlling portion 217 selects one from the frame data rates,which have been obtained by the cumulating portions Σ 215, 215, . . . ofthe pre-encoding process portion 103B for each quantizer scale Qn sothat the selected one does not exceed the maximum generated code amountper frame designated by the system controller 117 and is the closest tothe designated value. The quantizer scale (mquant) for each macro blockused in the quantizing device corresponding to the selected frame datarate is obtained from the normalized activity data norm_act placed inthe stream and supplied to a quantizing portion 218.

The quantizer scale for each macro block is placed as quantizer_scale onthe rear end side of the header portion of the stream as shown in FIG.16B and then sent to the quantizing portion 218.

The maximum generated code amount per frame is designated by for examplethe system controller 117 and supplied to the rate controlling portion217 through the CPU I/F block 221.

At that point, the value of the quantizer scale (mquant) for each macroblock can be decreased by one size in the range that does not exceed thedifference between the maximum generated code amount per frame, whichhas been designated by the system controller 117 and transferred throughthe CPU I/F block 221, and the generated code amount corresponding tothe quantizer scale (mquant) for each macro block, which has beenobtained from the normalized activity data norm_act placed in thestream. Thus, since a code amount close to the maximum generated codeamount designated by the system controller 117 and transferred troughthe CPU I/F block 221 is obtained, high picture quality can beaccomplished.

The quantizing portion 218 extracts the quantizer scale(quantizer_scale) designated by the rate controlling portion 217 in theforgoing manner from the stream and quantizes DCT coefficients with thequantizer table portion 211B corresponding to the extracted quantizerscale. At that point, since the quantizer scale supplied from the ratecontrolling portion 217 is the value of the quantizer scale (mquant)obtained from the normalized activity data norm_act, adaptivequantization is performed in consideration of visual characteristic.

The DCT coefficients quantized by the quantizing portion 218 are placedin the data portion of the stream as shown in FIG. 16C and supplied to aVLC portion 219. The DCT coefficients supplied to the VLC portion 219are scanned corresponding to for example the zigzag scanning method. Theresultant DCT coefficients are encoded with a variable length code withreference to a VLC table according to the two-dimensional Huffman code.The variable length code is bit-shifted so that it is byte-aligned andthen output as an MPEG ES.

At that point, as shown in FIG. 17, the header portion as the first halfportion of the stream is replaced with the MPEG header portion in whichthe MPEG header information of the slice layer or below is placed. Thevariable length code is placed in the data portion on the second halfside of the stream.

FIG. 18 shows an example of the structure for calculating a normalizedactivity norm_act according to the present invention. It is assumed thata normalized activity norm_act is calculated corresponding to Formula(8) and Formula (9).

As was described above, the activity portion 209 first obtains anactivity act. The activity act is supplied to first and second inputterminals of a multiplying device 301. The multiplying device 301calculates act×act. The calculated result is supplied to a first inputterminal of a multiplying device 311 through a register 304 and aregister 307. In addition, the calculated result is supplied to a firstinput terminal of an adding device 318 through a register 315.

An average activity avg_act that is output from the averaging portion206 is supplied to first and second input terminals of a multiplyingdevice 302. The multiplying device 302 calculates avg_act×avg_act. Thecalculated result is supplied to a first input terminal of a multiplyingdevice 312 through a register 305 and a register 308. In addition, thecalculated result is supplied to a first input terminal of an addingdevice 317 through a register 313.

An average activity avg_act that is output from the averaging portion206 is also supplied to a first input terminal of a multiplying device303. A parameter att is supplied to a second input terminal of themultiplying device 303. The parameter att is designated by for examplethe system controller 117 and supplied toe the multiplying device 303through a CPU I/F 221. The calculated result of the multiplying device303 is supplied to a first input terminal of an adding device 310through a register 306. A constant “1” is supplied to a second inputterminal of the adding device 310. The multiplying device 303 and theadding device 310 calculates att×avg_act+1 and obtains norm_gain. Theobtained result norm_gain is supplied to second input terminals of themultiplying device 311 and a multiplying device 312 through a register309.

The multiplying device 311 calculates norm_gain×(act×act). Thecalculated result is supplied to a second input terminal of the addingdevice 317 through a register 314. The adding device 317 calculatesavg_act×avg_act+norm_gain×(act×act). The calculated result is suppliedto a dividend input terminal of a dividing device 321 through a register319.

Likewise, the multiplying device 312 calculatesnorm_gain×(avg_act×avg_act). The calculated result is supplied to asecond input terminal of the adding device 318 through a register 316.The adding device 318 calculates norm_gain×(avg_act×avg_act)+act×act.The calculated result is supplied to a divisor input terminal of thedividing device 321 through a register 320.

The dividing device 321 calculates{avg_act×avg_act+norm_gain×(act×act)}÷{norm_gain×(avg_act×avg_act)+act×act}.In other words, an output of the dividing device 321 is a normalizedactivity norm_act according to an embodiment of the present invention.The normalized activity norm_act that is output from the multiplyingdevice 312 is output through a selector 325 and a register 324. In thestream, the normalized activity norm_act overwrites the activity actshown in FIG. 11C. The normalized activity norm_act is buried in thestream as shown in FIG. 12A and transferred to individual downstreamportions.

A comparing portion 322 monitors the value (divisor) supplied to theregister 320. When the divisor is “0”, for example a flag representingthat is supplied to the selector 325 through a register 326. An outputof the dividing device 321 is supplied to a first input terminal of theselector 325. A constant “1” is supplied to a second input terminal ofthe selector 325. When the flag supplied from the comparing portion 322represents that the divisor is “0”, the selector selects the constant“1” and outputs the normalized activity norm_act.

The forgoing example shows that the process of the MPEG encoder 103 isimplemented by hardware. However, according to the present invention,the process of the MPEG encoder 103 is not limited to such an example.In other words, the process of the MPEG encoder 103 can be implementedby software. In this case, for example, a computer apparatus is providedwith analog and digital interfaces for a video signal. Softwareinstalled on the computer is executed with a CPU and a memory. In theforgoing digital VTR structure, the CPU and memory may be substitutedfor the MPEG encoder 103.

The software is recorded as program data on a recording medium such as aCD-ROM (Compact Disc-Read Only Memory). The recording medium, on whichthe software has been recorded, is loaded to the computer apparatus.With a predetermined operation, the software is installed onto thecomputer apparatus. As a result, the process of the software can beexecuted. Since the structure of the computer apparatus is well known,its description will be omitted in the following.

FIG. 19 is a flow chart showing an example of the process of the MPEGencoder 103 implemented by software. Since the process of the flow chartis the same as the process implemented by the forgoing hardware, theprocess of the flow chart will be briefly described in consideration ofthe process implemented by hardware. Steps S1 to S7 correspond to theprocess of the forgoing input field activity averaging process portion103A. Steps S11 to S21 correspond to the process of the forgoingpre-encoding process portion 103B. Steps S31 to S38 correspond to theprocess of the forgoing encoding process portion 103C.

At step S1, the first step, video data is captured. At step S2, the nextstep, each header of the MPEG is extracted in the vertical blankingregion and stored in a memory. In other than the vertical blankingregion, the captured video data is stored in the memory.

At step S3, video data is converted from raster scan data into blockscan data. As a result, a macro block is extracted. This operation isperformed by controlling read addresses of the video data stored in thememory. At step S4, an activity of each macro block of the first fieldof the video data is calculated. At step S5, the obtained activitiesActivity (act) are cumulated and stored as the cumulated value sum tothe memory. The process from steps S3 to S5 is repeated until it hasbeen determined that the last macro block of the first field has beenprocessed at step S6. In other words, the cumulated value sum is the sumof activities of macro blocks for one field.

When it has been determined that the last macro block of one field hasbeen processed at step S6, the cumulated value stored in the memory isdivided by the number of macro blocks for one field at step S7. As aresult, the average value Activity (avg_act) of the field activities forthe field is obtained and stored in the memory.

When the average value Activity (avg_act) of the field activities hasbeen obtained, the flow advances to step S11. Like at step S3, at stepS11, the video data stored in the memory is converted from raster scandata into block scan data. As a result, a macro block is extracted. Atstep S12, the field DCT encoding mode or the frame DCT encoding mode isselected. The selected result is stored as DCT mode type data dct_typ inthe memory.

At step S13, with both the first and second fields, an activity act ofeach macro block is calculated. With the activities act and the averagevalue Activity (avg_act) of the field activities obtained and stored inthe memory at step S7, a normalized activity Activity (norm_act) of thepresent invention according to Formula (8) and Formula (9) is obtained.The obtained normalized activity Activity (norm_act) is stored in thememory.

The forgoing Formula (8) and Formula (9) can be expressed by softwareusing C language, which is a programming language. In the followingexpression, “MB_num” represents the total number of macro blocks of oneframe. In addition, a parameter att is constant. For example, att=0.125.

norm_gain = att × avg_act + 1 for (m = = 0; m < MB_num; m++ ) { if ((act[m] = = 0) && (avg_act = = 0)) norm_act [m] = 1 else norm_act [m] =((norm_gain × act [m] × act [m]) + (avg_act × avg_act)) ÷ ((act [m] ×act [m]) + (norm_gain × avg_act × avg_act)) } /* next macroblock */

At step S13, a normalized activity Activity (norm_act) is obtainedcorresponding to for example such a program.

At step S14, a macro block extracted from the video data at step S11 isdivided into DCT blocks each of which is composed of eight pixels×eightpixels. Each DCT block is two-dimensionally transformed in thetwo-dimensional DCT method. As a result, DCT coefficients are obtained.At step S15, the DCT coefficients are quantized with a quantizer table(quantizer_table). Thereafter, the flow advances to step S16.

The process from steps S16 to S20 is repeated with each quantizer scale(quantizer_scale) Qn value. As a result, the processes corresponding tothe forgoing Q_n portions 212, 212, . . . , the forgoing VLC portions213, 213, . . . , the forgoing cumulating portions Σ 214, 214, . . . ,and the forgoing cumulating portions Σ 215, 215, . . . In other words,at step S16, the DCT coefficients are quantized with quantizer scaleQ=1. At step S17, the DCT coefficients quantized with reference to theVLC table are encoded with a variable length code. At step S19, thegenerated code amount of the macro block with the variable length codeis calculated. At step S19, the generated code amount of each macroblock obtained at step S18 is cumulated for one frame. At step S20, itis determined whether or not there is another quantizer scale Qn. Whenit has been determined that there is another quantizer scale Qn, theflow returns to step S16. At step S16, the process for the otherquantizer scale Qn is performed. The generated code amountscorresponding to the individual quantizer scales Qn for one frame arestored in the memory.

When it has been determined that the cumulated value of the generatedcode amounts corresponding to all the quantizer scale values Qn for oneframe has been obtained at step S20, the flow advances to step S21. Atstep S21, it is determined whether or not the last macro block (MB) ofone frame has been processed. When the last macro block has not beenprocessed, the flow returns to step S11. When the last macro block hasbeen processed and the generated code amount for one frame has beenestimated, the flow advances to step S31. At step S31, the real encodingprocess is performed.

Like at S11, at step S31, video data stored in the memory is convertedfrom raster scan data into block scan data. As a result, a macro blockis extracted. At step S32, corresponding to the DCT mode type datadct_typ stored in the memory at step S12, the DCT encoding mode isdesignated.

At step S33, the macro block extracted from the video data at step S31is divided into DCT blocks each of which is composed of eightpixels×eight lines. Each DCT block is two-dimensionally transformedaccording to the two-dimensional DCT method. As a result, DCTcoefficients are obtained. The DCT coefficients are quantized with thequantizer table (quantizer_table) at step S34. Thereafter, the flowadvances to step S35.

At step S35, corresponding to the generated code amounts of thequantizer scales Qn, which have been estimated at steps S11 to S21, forone frame, the quantizer scales Qn used at step S36 are designated foreach macro block so as to control the generated code amount in the realencoding process.

Thereafter, the flow advances to step S36. With the quantizer scales Qndesignated at step S35, the DCT coefficients quantized with thequantizer table at step S34 are quantized. With reference to the VLCtable, the DCT coefficients quantized at step S36 are encoded with avariable length code at step S37. At step S38, it is determined whetheror not the last macro block of one frame has been processed. When it hasbeen determined that the last macro block of one frame has not beenprocessed, the flow returns to step S31. At step S31, the quantizingprocess and the variable length code encoding process are performed forthe next macro block. In contrast, when it has been determined that thelast macro block of one frame has been processed at step S38, theencoding process for one frame has been completed.

The forgoing example shows that the pre-encoding process from steps S11to S21 is different from the encoding process from steps S31 to S38.However, it should be noted that the present invention is not limited tosuch an example. For example, the generated code amounts estimated atsteps S11 to S21 are stored in the memory. Data that is obtained by thereal encoding process is selected from the stored data. As a result, theprocess from steps S31 to S38 can be contained as a loop in the processfrom steps S11 to S21.

In the forgoing description, the value of the parameter att is “0.125”.However, the present invention is not limited to such an example. Inother words, the parameter att may be another value. When the value ofthe parameter att is varied, the adaptive quantization can be performedcorresponding to various compression rates.

As was described above, according to the embodiment of the presentinvention, when a normalized activity is calculated, the normalizingrange is varied corresponding to a characteristic such as flatness andcomplexity of an objective picture—namely, an average activity of thepicture.

Thus, when a picture is wholly flat, the normalizing range of activitiesis narrowed. With activities that are almost equal in the whole picture,it can be quantized. Thus, with equal flatness in the whole picture, ahigh picture quality can be obtained.

In contrast, when a picture has both a complex region and a flat region,the normalizing range of activities is widened. A large code amount isassigned to a flat region so that it is finely quantized. A complexregion is coarsely quantized. As a result, a visually high qualitypicture can be obtained.

1. A picture processing apparatus, comprising: average activitycalculating means for calculating an average activity with picture data;setting means for dynamically setting a normalizing range so that thenormalizing range becomes continuously narrow after the average activitybecomes continuously small and dynamically setting a normalizing rangeso that the normalizing range becomes continuously wide after theaverage activity becomes continuously large; normalized activitycalculating means for calculating a normalized activity for each blockinto which the picture data is divided in the normalizing range set bysetting means according to the average activity and an activity for eachblock; and quantizing means for quantizing the picture data with thenormalized activity calculated by the normalized activity calculatingmeans.
 2. The picture processing apparatus as set forth in claim 1,wherein the normalized activity for each block is a value obtained bydividing the sum of the result of multiplying a coefficient proportionalto the average activity by the activity for each block and the averageactivity by the sum of the result of multiplying the coefficient by theaverage activity and the activity for each block.
 3. The pictureprocessing apparatus as set forth in claim 2, wherein when both thevalue of an activity of each block and the value of the average activityare 0, the normalized activity calculating means calculates the value ofthe normalized activity as
 1. 4. The picture processing apparatus as setforth in claim 1, wherein the normalized activity for each block is avalue obtained by dividing the sum of the result of multiplying acoefficient proportional to the average activity by the activity foreach block squared and the average activity squared by the sum of theresult of multiplying the coefficient by the average activity squaredand the activity for each block squared.
 5. The picture processingapparatus as set forth in claim 4, wherein when both the value of anactivity of each block and the value of the average activity are 0, thenormalized activity calculating means calculates the value of thenormalized activity as
 1. 6. The picture processing apparatus as setforth in claim 1, wherein the normalized activity calculating meanscalculates the average activity of one frame with the picture data ofthe first field of the one frame.
 7. A picture processing method,comprising the steps of: calculating an average activity with picturedata; dynamically setting a normalizing range so that the normalizingrange becomes continuously narrow after the average activity becomescontinuously small and dynamically setting a normalizing range so thatthe normalizing range becomes continuously wide after the averageactivity becomes continuously large; calculating a normalized activityfor each block into which the picture data is divided in the normalizingrange set by setting means according to the average activity and anactivity for each block; quantizing the picture data with the normalizedactivity calculated at the normalized activity calculating step.
 8. Apicture processing program for causing a computer apparatus to execute apicture processing method for quantizing picture data, the pictureprocessing method comprising the steps of: calculating an averageactivity with picture data; dynamically setting a normalizing range sothat the normalizing range becomes continuously narrow after the averageactivity becomes continuously small and dynamically setting anormalizing range so that the normalizing range becomes continuouslywide after the average activity becomes continuously large; calculatinga normalized activity for each block into which the picture data isdivided in the normalizing range set by setting means according to theaverage activity and an activity for each block; and quantizing thepicture data with the normalized activity calculated at the normalizedactivity calculating step.
 9. A recording medium on which pictureprocessing program for causing a computer apparatus to execute a pictureprocessing method for quantizing picture data is recorded, the pictureprocessing method comprising the steps of: calculating an averageactivity with picture data; dynamically setting a normalizing range sothat the normalizing range becomes continuously narrow after the averageactivity becomes continuously small and dynamically setting anormalizing range so that the normalizing range becomes continuouslywide after the average activity becomes continuously large; calculatinga normalized activity for each block into which the picture data isdivided in the normalizing range set by setting means according to theaverage activity and an activity for each block; and quantizing thepicture data with the normalized activity calculated at the normalizedactivity calculating step.
 10. A picture processing apparatus,comprising: an average activity calculating unit configured to calculatean average activity with picture data; a setting unit for dynamicallysetting a normalizing range so that the normalizing range becomescontinuously narrow after the average activity becomes continuouslysmall and dynamically setting a normalizing range so that thenormalizing range becomes continuously wide after the average activitybecomes continuously large; a normalized activity calculating unit forcalculating a normalized activity for each block into which the picturedata is divided in the normalizing range set by setting means accordingto the average activity and an activity for each block; a normalizedactivity calculating unit configured to dynamically designate anormalizing range according to the average activity calculated by theaverage activity calculating unit and calculating a normalized activitywith the average activity in the normalizing range; and a quantizingunit configured to quantize the picture data with the normalizedactivity calculated by the normalized activity calculating unit.
 11. Thepicture processing apparatus as set forth in claim 10, furthercomprising: an activity calculating unit for calculating activity foreach block.
 12. The picture processing apparatus as set forth in claim10, wherein the normalized activity for each block is a value obtainedby dividing the sum of the result of multiplying a coefficientproportional to the average activity by the activity for each block andthe average activity by the sum of the result of multiplying thecoefficient by the average activity and the activity for each block. 13.The picture processing apparatus as set forth in claim 12, wherein whenboth the value of an activity of each block and the value of the averageactivity are 0, the normalized activity calculating unit calculates thevalue of the normalized activity as
 1. 14. The picture processingapparatus as set forth in claim 10, wherein the normalized activity foreach block is a value obtained by dividing the sum of the result ofmultiplying a coefficient proportional to the average activity by theactivity for each block squared and the average activity squared by thesum of the result of multiplying the coefficient by the average activitysquared and the activity for each block squared.
 15. The pictureprocessing apparatus as set forth in claim 14, wherein when both thevalue of an activity of each block and the value of the average activityare 0, the normalized activity calculating unit calculates the value ofthe normalized activity as
 1. 16. The picture processing apparatus asset forth in claim 10, wherein the normalized activity calculating unitcalculates the average activity of one frame with the picture data ofthe first field of the one frame.
 17. The picture processing apparatusas set forth in claim 1, further comprising: activity calculating meansfor calculating activity for each block.